A policy gradient method for semi-Markov decision processes with application to call admission control

Solving a semi-Markov decision process (SMDP) using value or policy iteration requires precise knowledge of the probabilistic model and suffers from the curse of dimensionality. To overcome these limitations, we present a reinforcement learning approach where one optimizes the SMDP performance criterion with respect to a family of parameterised policies. We propose an online algorithm that simultaneously estimates the gradient of the performance criterion and optimises it using stochastic approximation. We apply our algorithm to call admission control. Index Terms Stochastic processes; Semi-Markov decision process; Policy gradient; Two-time scale; Call admission control. ∗Corresponding author. S. Singh is with the Signal Processing Group, Department of Engineering, Cambridge University, CB2 1PZ Cambridge, UK. Email: [email protected] Tel: +44 1223 332 784 Fax: +44 1223 332 662 V. Tadić is with the Department of Automatic Control and Systems Engineering, The University of Sheffield, S1 3JD Sheffield, UK. Email: [email protected] Tel: +44 114 222 5198 Fax: +44 (0)114 222 5661 A. Doucet is with the Signal Processing Group, Department of Engineering, Cambridge University, CB2 1PZ Cambridge, UK. Email: [email protected] Tel: +44 1223 332 676 Fax: +44 1223 332 662 February 14, 2005 DRAFT